{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas不为人知的八大实用技巧"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大家好，我今天勤快地回来了，这一期主要是和大家分享一些pandas的实用技巧，会在日常生活中大大提升效率，希望可以帮助到大家,还是老样子，\n",
    "先给大家奉上这一期的章节目录:\n",
    "\n",
    "1. 自定义pandas选项，设置\n",
    "2. 实用pandas中testing模块构建测试数据\n",
    "3. 巧用accessor接口方法\n",
    "4. 合并其他列拼接DatetimeIndex\n",
    "5. 使用分类数据（Categorical Data）节省时间和空间\n",
    "6. 利用Mapping巧妙实现映射\n",
    "7. 压缩pandas对象\n",
    "8. 源码及GitHub地址\n",
    "\n",
    "好啦，话不多说，让我们一个个看吧\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 自定义pandas选项，设置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，大家可能不知道，pandas里面有一个方法pd.set_option()，利用它我们可以改变一些pandas中默认的核心设置，\n",
    "从而适应我们自身的需要，开始前还是老样子，让我们先导入numpy和pandas包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Using pandas, Version 0.23.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "f'Using {pd.__name__}, Version {pd.__version__}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们编写一个start方法来实现自定义pandas设置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def start():\n",
    "    options = {\n",
    "        'display': {\n",
    "            'max_columns': None,\n",
    "            'max_colwidth': 25,\n",
    "            'expand_frame_repr': False,  # Don't wrap to multiple pages\n",
    "            'max_rows': 14,\n",
    "            'max_seq_items': 50,         # Max length of printed sequence\n",
    "            'precision': 4,\n",
    "            'show_dimensions': False\n",
    "        },\n",
    "        'mode': {\n",
    "            'chained_assignment': None   # Controls SettingWithCopyWarning\n",
    "        }\n",
    "    }\n",
    "\n",
    "    for category, option in options.items():\n",
    "        for op, value in option.items():\n",
    "            pd.set_option(f'{category}.{op}', value)  # Python 3.6+\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    start()\n",
    "    del start  # Clean up namespace in the interpreter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大家可以发现，我们在方法的最后调用了pandas的set_option方法，直接利用我们自定义的参数替代了原有的pandas参数，现在让我们测试一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_option('display.max_rows')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现max_rows 已经被替换成了我们设置的14，现在用一个真实的例子，我们利用一组公开的鲍鱼各项指标的数据来实验，数据源来自机器学习平台的公开数据\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>length</th>\n",
       "      <th>diam</th>\n",
       "      <th>height</th>\n",
       "      <th>weight</th>\n",
       "      <th>rings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>0.455</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.5140</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0.350</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.090</td>\n",
       "      <td>0.2255</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.420</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.6770</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>0.440</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.5160</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I</td>\n",
       "      <td>0.330</td>\n",
       "      <td>0.255</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>I</td>\n",
       "      <td>0.425</td>\n",
       "      <td>0.300</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.3515</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>F</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.415</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.7775</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4170</th>\n",
       "      <td>M</td>\n",
       "      <td>0.550</td>\n",
       "      <td>0.430</td>\n",
       "      <td>0.130</td>\n",
       "      <td>0.8395</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4171</th>\n",
       "      <td>M</td>\n",
       "      <td>0.560</td>\n",
       "      <td>0.430</td>\n",
       "      <td>0.155</td>\n",
       "      <td>0.8675</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4172</th>\n",
       "      <td>F</td>\n",
       "      <td>0.565</td>\n",
       "      <td>0.450</td>\n",
       "      <td>0.165</td>\n",
       "      <td>0.8870</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4173</th>\n",
       "      <td>M</td>\n",
       "      <td>0.590</td>\n",
       "      <td>0.440</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.9660</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4174</th>\n",
       "      <td>M</td>\n",
       "      <td>0.600</td>\n",
       "      <td>0.475</td>\n",
       "      <td>0.205</td>\n",
       "      <td>1.1760</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4175</th>\n",
       "      <td>F</td>\n",
       "      <td>0.625</td>\n",
       "      <td>0.485</td>\n",
       "      <td>0.150</td>\n",
       "      <td>1.0945</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4176</th>\n",
       "      <td>M</td>\n",
       "      <td>0.710</td>\n",
       "      <td>0.555</td>\n",
       "      <td>0.195</td>\n",
       "      <td>1.9485</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sex  length   diam  height  weight  rings\n",
       "0      M   0.455  0.365   0.095  0.5140     15\n",
       "1      M   0.350  0.265   0.090  0.2255      7\n",
       "2      F   0.530  0.420   0.135  0.6770      9\n",
       "3      M   0.440  0.365   0.125  0.5160     10\n",
       "4      I   0.330  0.255   0.080  0.2050      7\n",
       "5      I   0.425  0.300   0.095  0.3515      8\n",
       "6      F   0.530  0.415   0.150  0.7775     20\n",
       "...   ..     ...    ...     ...     ...    ...\n",
       "4170   M   0.550  0.430   0.130  0.8395     10\n",
       "4171   M   0.560  0.430   0.155  0.8675      8\n",
       "4172   F   0.565  0.450   0.165  0.8870     11\n",
       "4173   M   0.590  0.440   0.135  0.9660     10\n",
       "4174   M   0.600  0.475   0.205  1.1760      9\n",
       "4175   F   0.625  0.485   0.150  1.0945     10\n",
       "4176   M   0.710  0.555   0.195  1.9485     12"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = ('https://archive.ics.uci.edu/ml/'\n",
    "       'machine-learning-databases/abalone/abalone.data')\n",
    "cols = ['sex', 'length', 'diam', 'height', 'weight', 'rings']\n",
    "abalone = pd.read_csv(url, usecols=[0, 1, 2, 3, 4, 8], names=cols)\n",
    "abalone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看到，数据截断为14行，保留了小数点后4位小数作为精度，和我们刚刚设置的precision=4是一样的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 实用pandas中testing模块构建测试数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过pandas.util.testing提供的方法，我们可以很容易的通过几行代码就构建出一个简单的测试数据类型，比如我们现在构建一个DataTime类型的数据，\n",
    "时间间隔为月："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-01-31</th>\n",
       "      <td>0.3574</td>\n",
       "      <td>-0.8804</td>\n",
       "      <td>0.2669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-29</th>\n",
       "      <td>0.3775</td>\n",
       "      <td>0.1526</td>\n",
       "      <td>-0.4803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-03-31</th>\n",
       "      <td>1.3823</td>\n",
       "      <td>0.2503</td>\n",
       "      <td>0.3008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-04-30</th>\n",
       "      <td>1.1755</td>\n",
       "      <td>0.0785</td>\n",
       "      <td>-0.1791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-05-31</th>\n",
       "      <td>-0.9393</td>\n",
       "      <td>-0.9039</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 A       B       C\n",
       "2000-01-31  0.3574 -0.8804  0.2669\n",
       "2000-02-29  0.3775  0.1526 -0.4803\n",
       "2000-03-31  1.3823  0.2503  0.3008\n",
       "2000-04-30  1.1755  0.0785 -0.1791\n",
       "2000-05-31 -0.9393 -0.9039  1.1837"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas.util.testing as tm\n",
    "tm.N, tm.K = 15, 3         # 规定行和列\n",
    "\n",
    "import numpy as np\n",
    "np.random.seed(444)\n",
    "\n",
    "tm.makeTimeDataFrame(freq='M').head() # 设置时间间隔为月\n",
    "# tm.makeTimeDataFrame(freq='D').head()  设置时间间隔为天"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "瞎生成一组乱七八糟的数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>nTLGGTiRHF</th>\n",
       "      <td>-0.6228</td>\n",
       "      <td>0.6459</td>\n",
       "      <td>0.1251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WPBRn9jtsR</th>\n",
       "      <td>-0.3187</td>\n",
       "      <td>-0.8091</td>\n",
       "      <td>1.1501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7B3wWfvuDA</th>\n",
       "      <td>-1.9872</td>\n",
       "      <td>-1.0795</td>\n",
       "      <td>0.2987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yJ0BTjehH1</th>\n",
       "      <td>0.8802</td>\n",
       "      <td>0.7403</td>\n",
       "      <td>-1.2154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0luaYUYvy1</th>\n",
       "      <td>-0.9320</td>\n",
       "      <td>1.2912</td>\n",
       "      <td>-0.2907</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 A       B       C\n",
       "nTLGGTiRHF -0.6228  0.6459  0.1251\n",
       "WPBRn9jtsR -0.3187 -0.8091  1.1501\n",
       "7B3wWfvuDA -1.9872 -1.0795  0.2987\n",
       "yJ0BTjehH1  0.8802  0.7403 -1.2154\n",
       "0luaYUYvy1 -0.9320  1.2912 -0.2907"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.makeDataFrame().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于可以随机生成的数据类型, 一共大概有30多种，大家如果感兴趣可以多试试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['makeBoolIndex',\n",
       " 'makeCategoricalIndex',\n",
       " 'makeCustomDataframe',\n",
       " 'makeCustomIndex',\n",
       " 'makeDataFrame',\n",
       " 'makeDateIndex',\n",
       " 'makeFloatIndex',\n",
       " 'makeFloatSeries',\n",
       " 'makeIntIndex',\n",
       " 'makeIntervalIndex',\n",
       " 'makeMissingCustomDataframe',\n",
       " 'makeMissingDataframe',\n",
       " 'makeMixedDataFrame',\n",
       " 'makeMultiIndex',\n",
       " 'makeObjectSeries',\n",
       " 'makePanel',\n",
       " 'makePeriodFrame',\n",
       " 'makePeriodIndex',\n",
       " 'makePeriodPanel',\n",
       " 'makePeriodSeries',\n",
       " 'makeRangeIndex',\n",
       " 'makeStringIndex',\n",
       " 'makeStringSeries',\n",
       " 'makeTimeDataFrame',\n",
       " 'makeTimeSeries',\n",
       " 'makeTimedeltaIndex',\n",
       " 'makeUIntIndex',\n",
       " 'makeUnicodeIndex']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[i for i in dir(tm) if i.startswith('make')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这样我们如果有测试的需求，会很容易地构建相对应的假数据来测试。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 巧用accessor接口方法\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "accessor（访问器） 具体就是类似getter和setter，当然，Python里面不提倡存在setter和getter方法，但是这样可以便于大家理解，pandas Series类型有3类accessor：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'cat', 'dt', 'str'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series._accessors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- .cat用于分类数据，\n",
    " - .str用于字符串（对象）数据，\n",
    " - .dt用于类似日期时间的数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们从.str开始看：假设现在我们有一些原始的城市/州/ 邮编数据作为Dataframe的一个字段："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "addr = pd.Series([\n",
    "    'Washington, D.C. 20003',\n",
    "    'Brooklyn, NY 11211-1755',\n",
    "    'Omaha, NE 68154',\n",
    "    'Pittsburgh, PA 15211'\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     WASHINGTON, D.C. 20003\n",
       "1    BROOKLYN, NY 11211-1755\n",
       "2            OMAHA, NE 68154\n",
       "3       PITTSBURGH, PA 15211\n",
       "dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "addr.str.upper()  # 因为字符串方法是矢量化的，这意味着它们在没有显式for循环的情况下对整个数组进行操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    5\n",
       "1    9\n",
       "2    5\n",
       "3    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "addr.str.count(r'\\d')  # 查看邮编有几位"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们想把每一行分成城市，州，邮编分开，可以用正则；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>zip</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Washington</td>\n",
       "      <td>DC</td>\n",
       "      <td>20003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brooklyn</td>\n",
       "      <td>NY</td>\n",
       "      <td>11211-1755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Omaha</td>\n",
       "      <td>NE</td>\n",
       "      <td>68154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pittsburgh</td>\n",
       "      <td>PA</td>\n",
       "      <td>15211</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         city state         zip\n",
       "0  Washington    DC       20003\n",
       "1    Brooklyn    NY  11211-1755\n",
       "2       Omaha    NE       68154\n",
       "3  Pittsburgh    PA       15211"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = (r'(?P<city>[A-Za-z ]+), '      # One or more letters\n",
    "         r'(?P<state>[A-Z]{2}) '      # 2 capital letters\n",
    "         r'(?P<zip>\\d{5}(?:-\\d{4})?)')  # Optional 4-digit extension\n",
    "\n",
    "addr.str.replace('.', '').str.extract(regex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二个访问器.dt用于类似日期时间的数据。它其实属于Pandas的DatetimeIndex，如果在Series上调用，它首先转换为DatetimeIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   2018-03-31\n",
       "1   2018-06-30\n",
       "2   2018-09-30\n",
       "3   2018-12-31\n",
       "4   2019-03-31\n",
       "5   2019-06-30\n",
       "6   2019-09-30\n",
       "7   2019-12-31\n",
       "8   2020-03-31\n",
       "dtype: datetime64[ns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "daterng = pd.Series(pd.date_range('2018', periods=9, freq='Q'))  # 时间间隔为季度\n",
    "daterng"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Saturday\n",
       "1    Saturday\n",
       "2      Sunday\n",
       "3      Monday\n",
       "4      Sunday\n",
       "5      Sunday\n",
       "6      Monday\n",
       "7     Tuesday\n",
       "8     Tuesday\n",
       "dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "daterng.dt.day_name()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2   2018-09-30\n",
       "3   2018-12-31\n",
       "6   2019-09-30\n",
       "7   2019-12-31\n",
       "dtype: datetime64[ns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "daterng[daterng.dt.quarter > 2]  # 查看2019年第3季度和第4季度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3   2018-12-31\n",
       "7   2019-12-31\n",
       "dtype: datetime64[ns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "daterng[daterng.dt.is_year_end]  #查看年末的一天"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后有关.cat访问器我们会在第5个技巧中提到"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 合并其他列拼接DatetimeIndex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在先让我们构建一个包含时间类型数据的Dataframe："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.0767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.2798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0.4032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1.2377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.2060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.6187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.3786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.4730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>-2.1505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2016</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.6340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2016</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.7964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2016</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    year  month  day    data\n",
       "0   2017      1    1 -0.0767\n",
       "1   2017      1    2 -1.2798\n",
       "2   2017      1    3  0.4032\n",
       "3   2017      2    1  1.2377\n",
       "4   2017      2    2 -0.2060\n",
       "5   2017      2    3  0.6187\n",
       "6   2016      1    1  2.3786\n",
       "7   2016      1    2 -0.4730\n",
       "8   2016      1    3 -2.1505\n",
       "9   2016      2    1 -0.6340\n",
       "10  2016      2    2  0.7964\n",
       "11  2016      2    3  0.0005"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from itertools import product\n",
    "datecols = ['year', 'month', 'day']\n",
    "\n",
    "df = pd.DataFrame(list(product([2017, 2016], [1, 2], [1, 2, 3])),\n",
    "                  columns=datecols)\n",
    "df['data'] = np.random.randn(len(df))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以发现year，month，day是分开的三列，我们如果想要把它们合并为完整的时间并作为df的索引，可以这么做:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-01-01</th>\n",
       "      <td>2017</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.0767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-01-02</th>\n",
       "      <td>2017</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.2798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-01-03</th>\n",
       "      <td>2017</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0.4032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-02-01</th>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1.2377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-02-02</th>\n",
       "      <td>2017</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.2060</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            year  month  day    data\n",
       "2017-01-01  2017      1    1 -0.0767\n",
       "2017-01-02  2017      1    2 -1.2798\n",
       "2017-01-03  2017      1    3  0.4032\n",
       "2017-02-01  2017      2    1  1.2377\n",
       "2017-02-02  2017      2    2 -0.2060"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index = pd.to_datetime(df[datecols])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以扔掉没用的列并把这个df压缩为Series："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2017-01-01   -0.0767\n",
       "2017-01-02   -1.2798\n",
       "2017-01-03    0.4032\n",
       "2017-02-01    1.2377\n",
       "2017-02-02   -0.2060\n",
       "Name: data, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop(datecols, axis=1).squeeze()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'datetime64[ns]'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index.dtype_str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. 使用分类数据（Categorical Data）节省时间和空间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "刚刚我们在第3个技巧的时候提到了访问器，现在让我们来看最后一个.cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas中Categorical这个数据类型非常强大，通过类型转换可以让我们节省变量在内存占用的空间，提高运算速度，不过有关具体的pandas加速实战，我会在\n",
    "下一期说，现在让我们来看一个小栗子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    59\n",
       "1    59\n",
       "2    61\n",
       "3    59\n",
       "4    61\n",
       "5    53\n",
       "6    53\n",
       "7    59\n",
       "8    53\n",
       "9    53\n",
       "dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colors = pd.Series([\n",
    "    'periwinkle',\n",
    "    'mint green',\n",
    "    'burnt orange',\n",
    "    'periwinkle',\n",
    "    'burnt orange',\n",
    "    'rose',\n",
    "    'rose',\n",
    "    'mint green',\n",
    "    'rose',\n",
    "    'navy'\n",
    "])\n",
    "\n",
    "import sys\n",
    "colors.apply(sys.getsizeof)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们首先创建了一个Series，填充了各种颜色，接着查看了每个地址对应的颜色所占内存的大小\n",
    ">**注意这里我们使用sys.getsizeof()来获取占内存大小，但是实际上空格也是占内存的，sys.getsizeof('')返回的是49bytes**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们想把每种颜色用占内存更少的数字来表示（机器学习种非常常见），这样可以减少占用的内存，首先让我们创建一个mapper字典，给每一种颜色指定\n",
    "一个数字"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'periwinkle': 0, 'mint green': 1, 'burnt orange': 2, 'rose': 3, 'navy': 4}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapper = {v: k for k, v in enumerate(colors.unique())}\n",
    "mapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着我们把刚才的colors数组转化为int类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    1\n",
       "2    2\n",
       "3    0\n",
       "4    2\n",
       "5    3\n",
       "6    3\n",
       "7    1\n",
       "8    3\n",
       "9    4\n",
       "dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 也可以通过 pd.factorize(colors)[0] 实现\n",
    "as_int = colors.map(mapper)\n",
    "as_int"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再让我们看一下占用的内存："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    24\n",
       "1    28\n",
       "2    28\n",
       "3    24\n",
       "4    28\n",
       "5    28\n",
       "6    28\n",
       "7    28\n",
       "8    28\n",
       "9    28\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "as_int.apply(sys.getsizeof)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在可以观察到我们的内存占用的空间几乎是之前的一半，其实，刚刚我们做的正是模拟Categorical Data的转化原理。现在让我们直接调用一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "650"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colors.memory_usage(index=False, deep=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "495"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colors.astype('category').memory_usage(index=False, deep=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大家可能感觉节省的空间并不是非常大对不对？ 因为目前我们这个数据根本不是真实场景，我们仅仅把数据容量增加10倍，现在再让我们看看效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manycolors = colors.repeat(10)\n",
    "len(manycolors) / manycolors.nunique()  # Much greater than 2.0x "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Not using category : 6500'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"Not using category : { manycolors.memory_usage(index=False, deep=True)}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Using category : 585'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"Using category : { manycolors.astype('category').memory_usage(index=False, deep=True)}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这回内存的占用量差距明显就出来了，现在让我们用.cat来简化一下刚刚的工作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      periwinkle\n",
       "1      mint green\n",
       "2    burnt orange\n",
       "3      periwinkle\n",
       "4    burnt orange\n",
       "5            rose\n",
       "6            rose\n",
       "7      mint green\n",
       "8            rose\n",
       "9            navy\n",
       "dtype: category\n",
       "Categories (5, object): [burnt orange, mint green, navy, periwinkle, rose]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_colors = colors.astype('category')\n",
    "new_colors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['burnt orange', 'mint green', 'navy', 'periwinkle', 'rose'], dtype='object')"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_colors.cat.categories   # 可以使用.cat.categories查看代表的颜色"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们查看把颜色代表的数字："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3\n",
       "1    1\n",
       "2    0\n",
       "3    3\n",
       "4    0\n",
       "5    4\n",
       "6    4\n",
       "7    1\n",
       "8    4\n",
       "9    2\n",
       "dtype: int8"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_colors.cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们如果不满意顺序也可以从新排序："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    1\n",
       "2    2\n",
       "3    0\n",
       "4    2\n",
       "5    3\n",
       "6    3\n",
       "7    1\n",
       "8    3\n",
       "9    4\n",
       "dtype: int8"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_colors.cat.reorder_categories(mapper).cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有关cat其他的方法，我们还是可以通过遍历dir来查看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['add_categories',\n",
       " 'as_ordered',\n",
       " 'as_unordered',\n",
       " 'categories',\n",
       " 'codes',\n",
       " 'ordered',\n",
       " 'remove_categories',\n",
       " 'remove_unused_categories',\n",
       " 'rename_categories',\n",
       " 'reorder_categories',\n",
       " 'set_categories']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[i for i in dir(new_colors.cat) if not i.startswith('_')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Categorical 数据通常不太灵活，比如我们不能直接在new_colors上新增一个新的颜色，要首先通过\n",
    ".add_categories来添加"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'ccolors' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-36-1766a795336d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mccolors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0miloc\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m'a new color'\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'ccolors' is not defined"
     ]
    }
   ],
   "source": [
    "ccolors.iloc[5] = 'a new color'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_colors = new_colors.cat.add_categories(['a new color'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_colors.iloc[5] = 'a new color'  # 不会报错"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_colors.values  # 成功添加"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. 利用Mapping巧妙实现映射"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设现在我们有存贮国家的一组数据，和一组用来映射国家所对应的大洲的数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries = pd.Series([\n",
    "    'United States',\n",
    "    'Canada',\n",
    "    'Mexico',\n",
    "    'Belgium',\n",
    "    'United Kingdom',\n",
    "    'Thailand'\n",
    "])\n",
    "\n",
    "groups = {\n",
    "    'North America': ('United States', 'Canada', 'Mexico', 'Greenland'),\n",
    "    'Europe': ('France', 'Germany', 'United Kingdom', 'Belgium')\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以通过下面的方法来实现简单的映射："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any\n",
    "\n",
    "def membership_map(s: pd.Series, groups: dict,\n",
    "                   fillvalue: Any=-1) -> pd.Series:\n",
    "    # Reverse & expand the dictionary key-value pairs\n",
    "    groups = {x: k for k, v in groups.items() for x in v}\n",
    "    return s.map(groups).fillna(fillvalue)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    " membership_map(countries, groups, fillvalue='other')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很简单对不对，现在让我们看一下最关键的一行代码，groups = {x: k for k, v in groups.items() for x in v}，这个是我之前提到过的字典推导式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = dict(enumerate(('ab', 'cd', 'xyz')))\n",
    "{x: k for k, v in test.items() for x in v}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. 压缩pandas对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "如果你的pandas版本大于0.21.0，那么都可以直接把pandas用压缩形式写入，常见的类型有gzip, bz2, zip，这里我们直接用刚才鲍鱼的数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abalone.to_json('df.json.gz', orient='records',lines=True, compression='gzip')  # 压缩为gz类型\n",
    "abalone.to_json('df.json', orient='records', lines=True)                        #压缩为json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "os.path.getsize('df.json') / os.path.getsize('df.json.gz')  #压缩大小差了10倍，还是gz更厉害"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. 源码及GitHub地址"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这一期为大家总结了很多pandas实用的小技巧，希望大家喜欢\n",
    "\n",
    "我把这一期的ipynb文件和py文件放到了Github上，大家如果想要下载可以点击下面的链接：\n",
    " - Github仓库地址： [https://github.com/yaozeliang/pandas_share](https://github.com/yaozeliang/pandas_share)\n",
    "\n",
    "\n",
    "这一期就到这里啦，希望大家能够继续支持我，完结，撒花"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
